Loomio
March 23rd, 2016 08:41

setup backup

Pirate Praveen
Pirate Praveen Public Seen by 672

We can replicate postgresql and use rsync to backup uploads.
Replication howto: https://www.digitalocean.com/community/tutorials/how-to-set-up-master-slave-replication-on-postgresql-on-an-ubuntu-12-04-vps

We can use backup.poddery.com (we recently set it up for full backup of poddery.com). Any volunteers?

Raju Devidas

Raju Devidas March 23rd, 2016 11:48

Every time you ask for volunteers, thoughts come to my mind. We have mountains of work and small number of people to do it.

Pirate Praveen

Pirate Praveen March 24th, 2016 14:09

@isaagar thanks for volunteering to set this up. See the link above for steps to setup postgresql replication. Once you test replication between mahishasura.pxq.in and mahabali.pxq.in, we can give you access to diasp.in to setup the backup.

I Sagar

I Sagar March 24th, 2016 14:18

@praveenarimbrathod Thank you for providing opportunity.

Pirate Praveen

Pirate Praveen April 4th, 2016 05:01

I Sagar

I Sagar April 4th, 2016 05:07

@praveenarimbrathod , ok
I will update the status

Pirate Praveen

Pirate Praveen April 13th, 2016 06:36

@isaagar did you test it between two remote servers?

I Sagar

I Sagar April 14th, 2016 13:22

@praveenarimbrathod i don't have access to the test machines . Please , add me in the machines authorized users list.

I Sagar

I Sagar May 29th, 2016 07:06

We require a server for backup . As @praveenarimbrathod said one option is backup.poddery.com . Please , suggest any other option.

Pirate Praveen

Pirate Praveen May 30th, 2016 06:53

@isaagar I have asked poddery team about it. https://www.loomio.org/d/vojGH8by

We could add fosscommunity.in as a team member on diasp.in homepage.

I Sagar

I Sagar May 30th, 2016 07:14

ok , what is the size of data on diasp.in ?

Pirate Praveen

Pirate Praveen June 8th, 2016 16:13

@isaagar we can use backup.poddery.com. I'll share you access shortly.

I Sagar

I Sagar June 8th, 2016 16:14

@praveenarimbrathod ok

Pirate Praveen

Pirate Praveen June 8th, 2016 16:21

@isaagar follow these steps http://fosscommunity.in/index.php?title=Poddery#Setting_up_Backup and first create a similar plan for diasp.in at https://gitlab.com/piratemovin/diasp.in/wikis/home

  1. How much disk space is used now? Add disk used by db and uploads.
  2. We'll need to attach a new disk to backup.poddery.com and mount it like we do for poddery.com (get yourself familiar with LVM. How to create logical volumes and extend it)

Let me know if you have any questions.

I Sagar

I Sagar June 8th, 2016 17:14

@praveenarimbrathod i will start and let you know if any problem occurs

Pirate Praveen

Pirate Praveen August 29th, 2016 05:57

@isaagar where are we on it?

With the recent hard disk crash, we narrowly escaped losing data. We have to complete this urgently. @jayaura this is a prerequisite for updating diaspora. Can you help isaagar?

Aurabindo J

Aurabindo J August 29th, 2016 12:54

@praveenarimbrathod sure, please add @isaagar to diasp.in room

Pirate Praveen

Pirate Praveen August 29th, 2016 15:41

@jayaura I think @isaagar is in his hometown without good internet access. @rajudv do you know where is isaagar now? You can go through the link above and try setting up replication of postgres between two vms.

Pirate Praveen

Pirate Praveen September 7th, 2016 17:42

diasp.in has postgresql 9.3 and backup.fosscommunity.in (debian jessie) has postgresql 9.4. @vik @amardeepsingh can we have postgresql 9.4 backported to namaste?

Pirate Praveen

Pirate Praveen September 8th, 2016 10:57

I am creating an lxc container of hamara namaste locally. Here is the steps, if you want to try it https://gitlab.com/piratemovin/diasp.in/wikis/HamaraSetup

Pirate Praveen

Pirate Praveen September 9th, 2016 13:16

After a few trial and error, I got the hamara container to run. The problem was getting hamara-keyring package inside the rootfs. Now I have started backporting postgresql-9.4 from jessie.

Pirate Praveen

Pirate Praveen September 19th, 2016 04:00

As announced yesterday, https://poddery.com/posts/2639595 I will be updating postgres to 9.4 and then setting up replication.

I'll follow https://medium.com/@tk512/upgrading-postgresql-from-9-3-to-9-4-on-ubuntu-14-04-lts-2b4ddcd26535#.l0bk7ufl4

but use the backports repo I created at http://mahishasura.pxq.in/hamara/

Pirate Praveen

Pirate Praveen September 19th, 2016 05:15

Upgrade completed successfully. https://poddery.com/posts/2641176 Now going down for replication.

Pirate Praveen

Pirate Praveen September 19th, 2016 07:10

Replication setup failed as master and slave cannot be on different architectures https://poddery.com/posts/2641320 :(

Pirate Praveen

Pirate Praveen September 19th, 2016 08:33

Uploaded files are backed up using this cronjob now.

30 * * * * pgrep rsync || rsync -av --delete root@diasp.in:/usr/share/diaspora/public/uploads/ /var/lib/diasp.in/uploads/ >/var/lib/diasp.in/rsync-uploads.log

Pirate Praveen

Pirate Praveen September 19th, 2016 08:39

We may be able to use a different solution like http://www.pgbackrest.org/ any volunteers to try it?

Pirate Praveen

Pirate Praveen January 20th, 2017 13:55

We need to find another server with 64 bit Intel/AMD architecture to proceed. I recommend we go with a scaleway VC1S model (dual core 64 bit Intel VM with 2GB RAM). We can do a crowd funding to raise the amount, it comes to 2.99 Euro per month. If anyone has cheaper options to suggest, feel free.

@tvm @vik @amardeepsingh @akshay @fayadfami @anisha @dhanesh95

Dhanesh B. Sabane

Dhanesh B. Sabane January 20th, 2017 16:21

I'm a complete noob when it comes to servers, but I have a couple of friends and colleagues who might know something about it. If in case I get some useful information, I'll immediately share it here.

Pirate Praveen

Pirate Praveen January 20th, 2017 17:27

Everyone starts as a noob :) If there is interest you can master the servers.

Pirate Praveen

Pirate Praveen January 22nd, 2017 15:48

@amardeepsingh last time when we met at Baga, Goa, we talked about a backup server we can use. @isaagar can you check with Amardeep and confirm? If not, we'll proceed with scaleway.com option.

I Sagar

I Sagar January 26th, 2017 09:25

Talked to amardeep about the issue . He said if you want realtime backup then we should think of external server otherwise he's ok with daily backup using postgresqldump instead of slave setup.

Pirate Praveen

Pirate Praveen January 26th, 2017 09:40

Thanks for the update. I prefer slave setup as we get real-time incremental backups. With dump based approach, we may lose data after last dump in case of a failure.

Pirate Praveen

Pirate Praveen started a proposal January 26th, 2017 09:51

Buy a VC1S instance at scaleway.com for postgresql replication with crowd funding Closed 3:31pm - Thursday 2 Feb 2017

Outcome
by Pirate Praveen April 25th, 2017 05:31

We can get a VC1S from scaleway.com

We need a 64 bit Intel box to replicate postgresql database. scaleway.com offers the cheapest vps (poddery.com is using scaleway.com and it has been running great).

Starter VC1S: €2.99/month, 2 x86 64bit Cores, 2GB Memory, 50GB SSD Disk, 200Mbit/s Unmetered bandwidth

$ sudo du -hs /var/lib/postgresql/9.4/main
2.7G /var/lib/postgresql/9.4/main

Current DB size is 2.7 GB so this configuration would be sufficient.

Uploads are already backed up to backup.fosscommunity.in (which is running on an arm processor so we could do replicate postgresql there)

$ du -hs /usr/share/diaspora/public/uploads/27G /usr/share/diaspora/public/uploads/

We could share our backup server for git.fosscommunity.in as a return gift (which also share the same issue of 64 bit Intel to arm replication incompatibility).

Results
Agree - 3
Abstain - 3
Disagree - 3
Block - 3
3 people have voted (0%)
Pirate Praveen

Pirate Praveen
Agree
January 26th, 2017 09:51

yes, this is long overdue

I Sagar

I Sagar
Agree
January 26th, 2017 09:59

Dhanesh B. Sabane

Dhanesh B. Sabane
Agree
January 27th, 2017 03:56

Dhanesh B. Sabane

Dhanesh B. Sabane
Agree
January 27th, 2017 04:06

I'm slowly getting a hang of this.. Agreed! :thumbsup:

Pirate Praveen

Pirate Praveen February 3rd, 2017 10:02

I just signed up at scaleway.com. Access details are updated in our private repo, encrypted as usual (just added @isaagar ).

Pirate Praveen

Pirate Praveen February 3rd, 2017 10:13

@isaagar said he'll setup backup over the weekend.

[03:34:20 IST] ‎piratepraveen‎: isaagar, we now have a scaleway.com account
‎[03:34:38 IST] ‎piratepraveen‎: isaagar, would you be able to fins some time to complete backup?
‎[03:35:28 IST] ‎piratepraveen‎: I suggest we create one instance in Amsterdam, I guess Netherlands is better for privacy comapred to France
‎[03:35:46 IST] ‎piratepraveen‎: bsc, fayad dhanesh95 ^
‎[03:36:04 IST] ‎fayad‎: piratepraveen: hi
‎[03:37:14 IST] ‎isaagar‎: piratepraveen: i will do in this weekend

Pirate Praveen

Pirate Praveen February 3rd, 2017 17:16

[07:35:56  IST] ‎tvmanukrishnan‎: I have a dedi idling at online.net. Can spun up NAT ed VPS's. That would be good enough for backup, I hope. Additional IP with them cost 2 Euro + 3 Euro setup charges.
‎[07:36:32  IST] ‎tvmanukrishnan‎: Can have encrypted disks too.

@isaagar @fayadfami , we can use @tvm 's VPS then instead of scaleway.com

Manu Krishnan T V

Manu Krishnan T V February 9th, 2017 09:34

Do you need encrypted disks? If so, I need to create a KVM machine with disk encryption.

Otherwise, an LXC container will do, for which disk space can be expanded any time.

Btw, how much storage space are you looking for?

Pirate Praveen

Pirate Praveen February 9th, 2017 11:30

yes, we should encrypt the disk. postgresql data dir is about 2.7 GB. So I think 10GB disk would do for now.

Pirate Praveen

Pirate Praveen February 11th, 2017 08:43

@isaagar @fayadfami backup server is ready. You can find access details at https://gitlab.com/piratemovin/access

We should update https://gitlab.com/piratemovin/diasp.in/wikis/setting_up_backup as we configure the server so people know how to fix things if needed.

I Sagar

I Sagar February 12th, 2017 09:59

I am configuring mayasura and will update the wiki. @praveenarimbrathod Need help on server side configuration.

Pirate Praveen

Pirate Praveen February 12th, 2017 16:00

Feel free to ask here, xmpp, or irc/matrix.

I Sagar

I Sagar February 14th, 2017 07:12

@praveenarimbrathod @fayadfami @tvm On slave , it says could not connect to master server "210.7.64.54" .
We set max_wal_senders to 1 (on master) and including this backup server I think we actually have two backup servers . Should I increase value of max_wal_senders to 2 ? or Do you see any other problem ?

Pirate Praveen

Pirate Praveen February 14th, 2017 07:34

Since cannot use the first backup server, can't we remove the entries for the the first backup server?

I Sagar

I Sagar February 14th, 2017 17:30

Ok , I will comment out the entries of first server , to see if problem solves or not.

I Sagar

I Sagar February 16th, 2017 07:34

The problem was port 5432 of diasp.in was firewalled and after allowing port 5432 , slave is connecting to diasp.in.
But , I haven't seen in logs on slave machine info. relating live transactions . When I started slave machine then last entry in log says "ready to accept read only connections"
I think I should wait for some time if data is really replicating on slave.

I Sagar

I Sagar February 17th, 2017 05:56

I have checked running postgresql processes on master and slave server , it seems replication is running properly . I will update the wiki about the setup.
Also , how should we do the monitoring ??
@praveenarimbrathod how do you monitor backup server of poddery ?

Pirate Praveen

Pirate Praveen February 17th, 2017 07:03

@isaagar :thumbsup: super! For poddery, we just check if the backup server is running or not as backup server was getting hanged in the past. We can setup a ping test for the backup server. Also we should do a daily full db dump on the slave using cron job. We should keep the dump of last day and last week and delete the rest. This is just be on the safer side if some bad queries can mess up the db. Can you try that? or anyone else wants to volunteer for that?

Pirate Praveen

Pirate Praveen February 17th, 2017 07:04

Also please post an update about the backup setup from diasp.in diaspora account.

Manu Krishnan T V

Manu Krishnan T V February 19th, 2017 17:51

Just to confirm, are we using the NAT VPS I provided? If so, did you face any issues with the common ports not reaching the VPS?

I Sagar

I Sagar February 23rd, 2017 09:25

Unable to telnet port 11234 which is used by postgresql .

I Sagar

I Sagar February 23rd, 2017 09:26

Sorry for the delay , I will post about it after fixing replication.

I Sagar

I Sagar February 24th, 2017 06:22

I re-issued backup commands and replication is now working fine.

I Sagar

I Sagar March 12th, 2017 10:14

Added cronjob for full db dump .

I Sagar

I Sagar March 12th, 2017 10:25

@praveenarimbrathod Also , using uptimerobot I dont think its possible to check psql replication . I think we need to run custom script at master server for checking status..

Pirate Praveen

Pirate Praveen March 12th, 2017 11:21

With poddery.com, the slave server was going down unnoticed and breaking replication so added uptimerobot check to know if mysql slave is running. We can do a similar one here. Its not perfect monitoring, but better than not having any :) We can add more for sure.

I Sagar

I Sagar March 12th, 2017 13:52

In one case , I noticed that slave server was running at mayasura but with broken replication and I need to restart slave server to accept data segments from master server .
I think we need better solution for this case.

I Sagar

I Sagar April 8th, 2017 16:22

Still ,we haven't decided for the monitoring part . Zabbix is used at TBS for monitoring purposes which can solve our problem.

Pirate Praveen

Pirate Praveen April 9th, 2017 08:43

If we can use the same zabbix instance, we can configure zabbix for monitoring.

Pirate Praveen

Pirate Praveen July 2nd, 2018 19:54

@isaagar which server are we using for backup of uploads?

I Sagar

I Sagar July 6th, 2018 20:31

@praveenarimbrathod Uploads aren't backed up yet.I think @tvm provided extra disk on mayasura. Will try to do it over weekend

Dhanesh B. Sabane

Dhanesh B. Sabane July 21st, 2018 17:43

Do we need an action here? I'm ready to volunteer for the work. Will be needing some guidance though.