Loomio

setup backup

PP
Pirate Praveen Public Seen by 772

We can replicate postgresql and use rsync to backup uploads.
Replication howto: https://www.digitalocean.com/community/tutorials/how-to-set-up-master-slave-replication-on-postgresql-on-an-ubuntu-12-04-vps

We can use backup.poddery.com (we recently set it up for full backup of poddery.com). Any volunteers?

RD

Raju Devidas Wed 23 Mar 2016

Every time you ask for volunteers, thoughts come to my mind. We have mountains of work and small number of people to do it.

PP

Pirate Praveen Thu 24 Mar 2016

@isaagar thanks for volunteering to set this up. See the link above for steps to setup postgresql replication. Once you test replication between mahishasura.pxq.in and mahabali.pxq.in, we can give you access to diasp.in to setup the backup.

IS

I Sagar Thu 24 Mar 2016

@praveenarimbrathod Thank you for providing opportunity.

PP

Pirate Praveen Mon 4 Apr 2016

IS

I Sagar Mon 4 Apr 2016

@praveenarimbrathod , ok
I will update the status

PP

Pirate Praveen Wed 13 Apr 2016

@isaagar did you test it between two remote servers?

IS

I Sagar Thu 14 Apr 2016

@praveenarimbrathod i don't have access to the test machines . Please , add me in the machines authorized users list.

IS

I Sagar Sun 29 May 2016

We require a server for backup . As @praveenarimbrathod said one option is backup.poddery.com . Please , suggest any other option.

PP

Pirate Praveen Mon 30 May 2016

@isaagar I have asked poddery team about it. https://www.loomio.org/d/vojGH8by

We could add fosscommunity.in as a team member on diasp.in homepage.

IS

I Sagar Mon 30 May 2016

ok , what is the size of data on diasp.in ?

PP

Pirate Praveen Wed 8 Jun 2016

@isaagar we can use backup.poddery.com. I'll share you access shortly.

IS

I Sagar Wed 8 Jun 2016

@praveenarimbrathod ok

PP

Pirate Praveen Wed 8 Jun 2016

@isaagar follow these steps http://fosscommunity.in/index.php?title=Poddery#Setting_up_Backup and first create a similar plan for diasp.in at https://gitlab.com/piratemovin/diasp.in/wikis/home

  1. How much disk space is used now? Add disk used by db and uploads.
  2. We'll need to attach a new disk to backup.poddery.com and mount it like we do for poddery.com (get yourself familiar with LVM. How to create logical volumes and extend it)

Let me know if you have any questions.

IS

I Sagar Wed 8 Jun 2016

@praveenarimbrathod i will start and let you know if any problem occurs

PP

Pirate Praveen Mon 29 Aug 2016

@isaagar where are we on it?

With the recent hard disk crash, we narrowly escaped losing data. We have to complete this urgently. @jayaura this is a prerequisite for updating diaspora. Can you help isaagar?

AJ

Aurabindo J Mon 29 Aug 2016

@praveenarimbrathod sure, please add @isaagar to diasp.in room

PP

Pirate Praveen Mon 29 Aug 2016

@jayaura I think @isaagar is in his hometown without good internet access. @rajudv do you know where is isaagar now? You can go through the link above and try setting up replication of postgres between two vms.

PP

Pirate Praveen Wed 7 Sep 2016

diasp.in has postgresql 9.3 and backup.fosscommunity.in (debian jessie) has postgresql 9.4. @vik @amardeepsingh can we have postgresql 9.4 backported to namaste?

PP

Pirate Praveen Thu 8 Sep 2016

I am creating an lxc container of hamara namaste locally. Here is the steps, if you want to try it https://gitlab.com/piratemovin/diasp.in/wikis/HamaraSetup

PP

Pirate Praveen Fri 9 Sep 2016

After a few trial and error, I got the hamara container to run. The problem was getting hamara-keyring package inside the rootfs. Now I have started backporting postgresql-9.4 from jessie.

PP

Pirate Praveen Mon 19 Sep 2016

As announced yesterday, https://poddery.com/posts/2639595 I will be updating postgres to 9.4 and then setting up replication.

I'll follow https://medium.com/@tk512/upgrading-postgresql-from-9-3-to-9-4-on-ubuntu-14-04-lts-2b4ddcd26535#.l0bk7ufl4

but use the backports repo I created at http://mahishasura.pxq.in/hamara/

PP

Pirate Praveen Mon 19 Sep 2016

Upgrade completed successfully. https://poddery.com/posts/2641176 Now going down for replication.

PP

Pirate Praveen Mon 19 Sep 2016

Replication setup failed as master and slave cannot be on different architectures https://poddery.com/posts/2641320 :(

PP

Pirate Praveen Mon 19 Sep 2016

Uploaded files are backed up using this cronjob now.

30 * * * * pgrep rsync || rsync -av --delete root@diasp.in:/usr/share/diaspora/public/uploads/ /var/lib/diasp.in/uploads/ >/var/lib/diasp.in/rsync-uploads.log

PP

Pirate Praveen Mon 2 Jul 2018

@isaagar which server are we using for backup of uploads?

PP

Pirate Praveen Mon 19 Sep 2016

We may be able to use a different solution like http://www.pgbackrest.org/ any volunteers to try it?

PP

Pirate Praveen Fri 20 Jan 2017

We need to find another server with 64 bit Intel/AMD architecture to proceed. I recommend we go with a scaleway VC1S model (dual core 64 bit Intel VM with 2GB RAM). We can do a crowd funding to raise the amount, it comes to 2.99 Euro per month. If anyone has cheaper options to suggest, feel free.

@tvm @vik @amardeepsingh @akshay @fayadfami @anisha @dhanesh95

DBS

Dhanesh B. Sabane Fri 20 Jan 2017

I'm a complete noob when it comes to servers, but I have a couple of friends and colleagues who might know something about it. If in case I get some useful information, I'll immediately share it here.

PP

Pirate Praveen Fri 20 Jan 2017

Everyone starts as a noob :) If there is interest you can master the servers.

PP

Pirate Praveen Sun 22 Jan 2017

@amardeepsingh last time when we met at Baga, Goa, we talked about a backup server we can use. @isaagar can you check with Amardeep and confirm? If not, we'll proceed with scaleway.com option.

IS

I Sagar Thu 26 Jan 2017

Talked to amardeep about the issue . He said if you want realtime backup then we should think of external server otherwise he's ok with daily backup using postgresqldump instead of slave setup.

PP

Pirate Praveen Thu 26 Jan 2017

Thanks for the update. I prefer slave setup as we get real-time incremental backups. With dump based approach, we may lose data after last dump in case of a failure.

PP

Pirate Praveen started a proposal Thu 26 Jan 2017

Buy a VC1S instance at scaleway.com for postgresql replication with crowd funding Closed Thu 2 Feb 2017

Outcome
by Pirate Praveen Tue 25 Apr 2017

We can get a VC1S from scaleway.com

We need a 64 bit Intel box to replicate postgresql database. scaleway.com offers the cheapest vps (poddery.com is using scaleway.com and it has been running great).

Starter VC1S: €2.99/month, 2 x86 64bit Cores, 2GB Memory, 50GB SSD Disk, 200Mbit/s Unmetered bandwidth

$ sudo du -hs /var/lib/postgresql/9.4/main
2.7G /var/lib/postgresql/9.4/main

Current DB size is 2.7 GB so this configuration would be sufficient.

Uploads are already backed up to backup.fosscommunity.in (which is running on an arm processor so we could do replicate postgresql there)

$ du -hs /usr/share/diaspora/public/uploads/27G /usr/share/diaspora/public/uploads/

We could share our backup server for git.fosscommunity.in as a return gift (which also share the same issue of 64 bit Intel to arm replication incompatibility).

Agree - 3
Abstain - 0
Disagree - 0
Block - 0
3 people have voted (8%)
PP

Pirate Praveen
Agree
Thu 26 Jan 2017

yes, this is long overdue

DBS

Dhanesh B. Sabane
Agree
Fri 27 Jan 2017

I'm slowly getting a hang of this.. Agreed! :thumbsup:

PP

Pirate Praveen Fri 3 Feb 2017

I just signed up at scaleway.com. Access details are updated in our private repo, encrypted as usual (just added @isaagar ).

PP

Pirate Praveen Fri 3 Feb 2017

@isaagar said he'll setup backup over the weekend.

[03:34:20 IST] ‎piratepraveen‎: isaagar, we now have a scaleway.com account
‎[03:34:38 IST] ‎piratepraveen‎: isaagar, would you be able to fins some time to complete backup?
‎[03:35:28 IST] ‎piratepraveen‎: I suggest we create one instance in Amsterdam, I guess Netherlands is better for privacy comapred to France
‎[03:35:46 IST] ‎piratepraveen‎: bsc, fayad dhanesh95 ^
‎[03:36:04 IST] ‎fayad‎: piratepraveen: hi
‎[03:37:14 IST] ‎isaagar‎: piratepraveen: i will do in this weekend

PP

Pirate Praveen Fri 3 Feb 2017

[07:35:56  IST] ‎tvmanukrishnan‎: I have a dedi idling at online.net. Can spun up NAT ed VPS's. That would be good enough for backup, I hope. Additional IP with them cost 2 Euro + 3 Euro setup charges.
‎[07:36:32  IST] ‎tvmanukrishnan‎: Can have encrypted disks too.

@isaagar @fayadfami , we can use @tvm 's VPS then instead of scaleway.com

MKT

Manu Krishnan T V Thu 9 Feb 2017

Do you need encrypted disks? If so, I need to create a KVM machine with disk encryption.

Otherwise, an LXC container will do, for which disk space can be expanded any time.

Btw, how much storage space are you looking for?

PP

Pirate Praveen Thu 9 Feb 2017

yes, we should encrypt the disk. postgresql data dir is about 2.7 GB. So I think 10GB disk would do for now.

PP

Pirate Praveen Sat 11 Feb 2017

@isaagar @fayadfami backup server is ready. You can find access details at https://gitlab.com/piratemovin/access

We should update https://gitlab.com/piratemovin/diasp.in/wikis/setting_up_backup as we configure the server so people know how to fix things if needed.

IS

I Sagar Sun 12 Feb 2017

I am configuring mayasura and will update the wiki. @praveenarimbrathod Need help on server side configuration.

PP

Pirate Praveen Sun 12 Feb 2017

Feel free to ask here, xmpp, or irc/matrix.

IS

I Sagar Tue 14 Feb 2017

@praveenarimbrathod @fayadfami @tvm On slave , it says could not connect to master server "210.7.64.54" .
We set max_wal_senders to 1 (on master) and including this backup server I think we actually have two backup servers . Should I increase value of max_wal_senders to 2 ? or Do you see any other problem ?

PP

Pirate Praveen Tue 14 Feb 2017

Since cannot use the first backup server, can't we remove the entries for the the first backup server?

IS

I Sagar Tue 14 Feb 2017

Ok , I will comment out the entries of first server , to see if problem solves or not.

IS

I Sagar Thu 16 Feb 2017

The problem was port 5432 of diasp.in was firewalled and after allowing port 5432 , slave is connecting to diasp.in.
But , I haven't seen in logs on slave machine info. relating live transactions . When I started slave machine then last entry in log says "ready to accept read only connections"
I think I should wait for some time if data is really replicating on slave.

IS

I Sagar Fri 17 Feb 2017

I have checked running postgresql processes on master and slave server , it seems replication is running properly . I will update the wiki about the setup.
Also , how should we do the monitoring ??
@praveenarimbrathod how do you monitor backup server of poddery ?

PP

Pirate Praveen Fri 17 Feb 2017

@isaagar :thumbsup: super! For poddery, we just check if the backup server is running or not as backup server was getting hanged in the past. We can setup a ping test for the backup server. Also we should do a daily full db dump on the slave using cron job. We should keep the dump of last day and last week and delete the rest. This is just be on the safer side if some bad queries can mess up the db. Can you try that? or anyone else wants to volunteer for that?

IS

I Sagar Sun 12 Mar 2017

Added cronjob for full db dump .

PP

Pirate Praveen Fri 17 Feb 2017

Also please post an update about the backup setup from diasp.in diaspora account.

IS

I Sagar Thu 23 Feb 2017

Sorry for the delay , I will post about it after fixing replication.

MKT

Manu Krishnan T V Sun 19 Feb 2017

Just to confirm, are we using the NAT VPS I provided? If so, did you face any issues with the common ports not reaching the VPS?

IS

I Sagar Thu 23 Feb 2017

Unable to telnet port 11234 which is used by postgresql .

IS

I Sagar Fri 24 Feb 2017

I re-issued backup commands and replication is now working fine.

IS

I Sagar Sun 12 Mar 2017

@praveenarimbrathod Also , using uptimerobot I dont think its possible to check psql replication . I think we need to run custom script at master server for checking status..

PP

Pirate Praveen Sun 12 Mar 2017

With poddery.com, the slave server was going down unnoticed and breaking replication so added uptimerobot check to know if mysql slave is running. We can do a similar one here. Its not perfect monitoring, but better than not having any :) We can add more for sure.

IS

I Sagar Sun 12 Mar 2017

In one case , I noticed that slave server was running at mayasura but with broken replication and I need to restart slave server to accept data segments from master server .
I think we need better solution for this case.

IS

I Sagar Sat 8 Apr 2017

Still ,we haven't decided for the monitoring part . Zabbix is used at TBS for monitoring purposes which can solve our problem.

PP

Pirate Praveen Sun 9 Apr 2017

If we can use the same zabbix instance, we can configure zabbix for monitoring.

IS

I Sagar Fri 6 Jul 2018

@praveenarimbrathod Uploads aren't backed up yet.I think @tvm provided extra disk on mayasura. Will try to do it over weekend

DBS

Dhanesh B. Sabane Sat 21 Jul 2018

Do we need an action here? I'm ready to volunteer for the work. Will be needing some guidance though.