database backup

Pirate Praveen
Pirate Praveen Public Seen by 294

We need to back up database and Nitesh had volunteered to set it up.

Pirate Praveen

Pirate Praveen September 18th, 2013 13:12

@niteshajain any updates?

Manu Krishnan T V

Manu Krishnan T V September 28th, 2013 17:33

I am tired of following discussions at multiple platforms. Feels like this topic made few progress through the closed mails between few members.

Nitesh A Jain

Nitesh A Jain October 10th, 2013 13:19

So i have taken a backup of the DB and the pictures and it is currently on the server itself. We need a place where we can safely store this backup...

Pirate Praveen

Pirate Praveen October 26th, 2013 07:25

@manukrishnantv did we use Sooraj's sever for this? What is the current status?

Manu Krishnan T V

Manu Krishnan T V October 26th, 2013 09:30

Sooraj's server is not yet used for backup. It has a openmeetings instance running and also configured to deliver sooraj's website. If openmeetings is not in need, I can stop that.

Pirate Praveen

Pirate Praveen March 21st, 2014 17:25

@niteshajain @manukrishnantv I think we should set it up soon. How about meeting up one day for this? irc/offline.

Nitesh A Jain

Nitesh A Jain March 21st, 2014 18:46

Saturday or Sunday is fine with me. Pinging @vamseekanakala also

Pirate Praveen

Pirate Praveen March 31st, 2014 06:05

@shafeeqk can you work with @niteshajain and @vamseekanakala to setup backup? @manukrishnantv can give access to the backup server.

Shafeeq K

Shafeeq K March 31st, 2014 14:36

@praveenarimbrathod Sure. What should I do first?

Pirate Praveen

Pirate Praveen March 31st, 2014 14:40

@shafeeqk try backing up your pod first.

Manu Krishnan T V

Manu Krishnan T V April 2nd, 2014 04:19

@praveenarimbrathod @shafeek I believe a Master/Slave setup will be best option to backup the database, which is the most important part of a pod. Poddery database was around 5GB when we migrated to Knightswarm. So, dumping it all the time and moving to a different location doesn't seems to be a viable option.

Also, files can be backed up using tools like duplicity, which supports incremental backups through rsync and other protocols.

Pirate Praveen

Pirate Praveen April 3rd, 2014 11:30

@shafeeqk we should ensure the data is encrypted on transport (mysql replication has that option) as well as storage (replication server should store its data on an encrypted partition.

Pirate Praveen

Pirate Praveen April 3rd, 2014 11:33

Shafeeq K

Shafeeq K April 3rd, 2014 14:51

@praveenarimbrathod I see. I backed up my pod database using mysqldump. I'll try setting up a slave for it and using SSH Tunnel. But for encrypted storage, don't we need a host that allows full disk encryption? Also, @manukrishnantv , should we detach the slave after it is set up to make sure that the data stays intact?

Pirate Praveen

Pirate Praveen April 4th, 2014 06:51

@shafeeqk , Sooraj has one server with full disk encryption setup. Manu has the access details.

Manu Krishnan T V

Manu Krishnan T V April 5th, 2014 04:15

@shafeeqk I'll be sitting with Sooraj today for fixing up few things in that server. Will share the details after that.

Lets create a script to detach the server for a while to create a SQL dump and reattach again.

Manu Krishnan T V

Manu Krishnan T V August 23rd, 2014 16:29

Its high time we look into this seriously, as we are taking some decisions which might not be agreed by the governmental bodies, like the take on Islamic State creating an account in our pod (Still on voting).

Pirate Praveen

Pirate Praveen February 10th, 2016 09:46

@manukrishnantv we have to revisit it. What is the status of Sooraj's server?

Manu Krishnan T V

Manu Krishnan T V February 16th, 2016 14:34

The server is still up and running, but hosts Varnam build using Jenkins, and its log eats up lot of space. Last time when I checked, it had around 20GB free out of 80.

Pirate Praveen

Pirate Praveen March 14th, 2016 06:56

@manukrishnantv I think it is better to get a cheap scaleway vps at 2.99 euros. We can backup diasp.in as well there.

Pirate Praveen

Pirate Praveen started a proposal March 14th, 2016 07:00

Buy a cheap vps for backup Closed 12:07pm - Sunday 20 Mar 2016

by Pirate Praveen April 25th, 2017 05:22

Going ahead with creating backup on a scaleway vps.

We can get one for 2.99 euro per month https://www.scaleway.com/pricing/ This way we will have full control over the backup system as well. We can share it with diasp.in as well.

Agree - 1
Abstain - 1
Disagree - 1
Block - 1
-39 people have voted (-3900%)
Pirate Praveen

Pirate Praveen
March 14th, 2016 07:00

We've been dragging this for long, lets do it

Manu Krishnan T V

Manu Krishnan T V March 14th, 2016 07:18

I haven't tried Online.net's services yet, but they have been there for a while and offer cheaper servers.

The 2.99 euro server comes with a 50GB SSD. Additional volumes can be mounted at 1 Euro per 50GB, which seems cheap.

Meanwhile, a 100GB on Amazon S3, will cost just above 2$, on pay as you go. So, there are alternatives.

Pirate Praveen

Pirate Praveen March 14th, 2016 07:21

@manukrishnantv but Amazon is US based, it doesn't make sense to backup in US when we specifically chose non-US provider for main service.

Pirate Praveen

Pirate Praveen March 20th, 2016 13:47

Finally after a whole day's effort we have full backup of poddery.com

  1. backup.poddery.com is running on a Scaleway.com C1 VPS (arm soc).
  2. /var/lib/diaspora is a 50GB encrypted disk
  3. mysql replication uses an ssh tunnel to poddery.com so replication is encrypted. sshtunnel user is created on both servers. Can someone create a systemd service to start this tunnel automatically before mysql?
  4. an hourly cron job will rsync diaspora/public/uploads to /var/lib/diaspora/uploads.

http://fosscommunity.in/index.php?title=Poddery has some initial documentation. Someone please verify the backup is working and ask me if the docs are not clear on any point.